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rj- This qualitative study describes strategies employed by sophisticated adult World Wide Web users 

^ as they evaluate authentic Web information with the purpose of adapting these strategies for children in K- 
12 settings. The participants in this study followed think-aloud protocols and answered interview questions 
^ about two Web documents containing numerous misinformation devices. Evaluative strategies from these 
p verbalizations were extracted and analyzed. Findings include a list of strategies and a description of three 

pj evaluative “styles.” Finally, suggestions for the use and teaching of these strategies in elementary school 
through middle school are made. 

INTRODUCTION 

As World Wide Web access expands into schools and homes, children will likely encounter the 
misinformation often found in this medium. Are children, often alone and unsupervised, equipped to sort 
good information from bad? What specific vulnerabilities and dangers do they face? How can educators 
equip them with the evaluative skills they need to sift through this new wealth of information?, Purpose 

The literature establishes misinformation as a potential problem for Internet users (Neavill, 1993; 

Viehland, 1993). The primary cause of this situation is an almost universal lack of gatekeeping and central 
authority. As a safeguard against nuclear attack, the Internet was built without a centralized controlling 
authority to distribute the information base throughout the United States (Stoker & Cooke, 1995). Because 
of this lack of control “anybody can publish anything.” (Neavill, 1984, p. 87) Without editorial control, 
documents flawed by bias, mistakes, lies, scholarly misconduct, or any of a number of other flaws, can 
circulate freely and instantly. Two related but less important causes of the misinformation problem are 
anonymity and hacking. Because the Internet allows anonymity, authors publishing misinformation fear no 
reprisals. 

Hackers often break into government servers, intercepting, interrupting, and changing data without 
discovery (Stephens, 1995). If hackers can access data, they can change it. University servers, the most 
complete repositories of online information, are particularly vulnerable to hacker attack because of 
inadequate preventive measures (Coutorie, 1995). 

Although censorship and technological screening have been suggested as possible solutions to the 
misinformation problem, the most practical approach is for readers to evaluate Web information 
themselves (Breivik, Senn, & Gee, 1989). This issue is a practical as well as a philosophical one. It is 
unlikely that professional information managers can keep pace with new information on the Web in their 
efforts to index, evaluate, and screen information. Screening software, while effective to a degree, cannot 
filter out all objectionable material or misinformation. Even if these measures were totally effective, the 
philosophical and legal issues of freedom of speech and individual interpretations of truth present 
insurmountable dilemmas. For now, Internet users must recognize the need to do their own sifting and ' / 
evaluating of Web information. ' 
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Are Internet users equipped with the skills they need for such a task? Critical skills to evaluate Web 
information may or may not be similar to those needed for general information literacy. Little is known 
about information evaluation in any medium, and existent studies indicate potential difficulties. Grice 
(1975) theorizes that people tend to believe that most information is true, and this theory harmonizes with 
Gilovich's assertion that people do not evaluate all incoming information for efficiency reasons (1991). At 
least two researchers have noted a remarkably passive acceptance of misinformation in memory studies 
(Belli, 1989; Highhouse & Bottrill, 1995). Analyses Studies of scholarly misconduct demonstrate that have 
highlighted this problem in even the supposedly discerning academic community is prone to evaluative 
passivity arenas (Kochan & Budd, 1992). Others writers have compiled case studies and long catalogs of 
successful hoaxes in the mass media hoaxes (Bird, 1996; Fedler, 1989; Tamarkin, 1993). 

A review of psychological and social psychological literature reveals several disturbing possibilities 
about the effects of misinformation. People seem highly vulnerable to the manipulation of superficial 
presentation characteristics. An illustration of this fact is that subjects tend to accept without question 
information presented by a person perceived as having high status or expertise without question (Petty & 
Cacioppo, 1986). In addition, readers rate texts with greater numbers of messages as being more believable 
than texts with fewer messages, regardless of message quality (Petty & Cacioppoe, 1984). Audience 
members are more likely to accept arguments greeted with the enthusiasm of their fellow listeners (Axsom, 
Yates, & Chaiken, 1987). In addition to these flaws in evaluative practice, people are vulnerable to the 
effects of misinformation may have disquieting manipulative effects upon memory. Loftus (1975) initiated 
a major trend in memory research with her discovery that witnesses of complex events exposed to 
conflicting misinformation after the event often reported the misinformation as part of the original memory. 
•Anderson (1965) established that people tend to believe information presented first in a sequence, and 
disregard conflicting information presented later. These fragmentary and contradictory glimpses of’how 
misjudgment and misinformation affects critical judgment and memory illustrate the fact that the poorly 
understood reader-misinformation interaction can potentially have profound results. 

The few scholars who have studied misinformation in telecommunicated contexts report interesting 
findings. Aycock and Buchignani (1995), Hemon, (1995), and Viehland (1993)found a passive acceptance 
to misinformation in the electronic medium similar to that found by psychologists in other media. Sachs 
(1995) noted how the online discussion group he studied tended to reinforce the political biases of its 
readers. In addition, Aycock and Buchignani, as well as and Viehland, remarked upon how quickly and 
widely telecommunicated misinformation spread in their analyses of authentic online misinformation cases. 

Several scholars and practitioners have published skill sets for online information evaluation (see, 
for example, Stripling & Pitts, 1988; Weisburg & Toor, 1994). Hemon (1995) asserts that Internet 
information is similar to information in other media in terms of quality. Thus, it is reasonable to assume 
that evaluative skill needed for online information differ little from skills needed for information in more 
traditional formats. However, this assumption has not been adequately tested. Also, as the literature shows, 
our knowledge of the thinking processes of readers as they encounter misinformation is fragmentary. One 
possible approach to this lack of knowledge is to examine the strategies of experts as they evaluate 
information. The purpose of this study is to describe the strategies used by sophisticated Web users as they 
make critical judgments about the quality of information found in authentic Web documents, and to adapt 
these strategies for use in K-12 settings. 

For the purposes of this study, “misinformation” is defined as material presented as true although 
. it contradicts facts presented in standard reference works. Information quality literature describes how 
authors can misrepresent facts through an array of linguistic tactics (see, for example, Lazere, 1982). These 
misrepresentations relate directly in some cases to the presentation manipulation findings described above. 
■Riese linguistic tactics are labeled here as “devices of misinformation.” The term “trigger” refers to the 
linguistic signal that marks the presence of a device. An information “problem” is a shortcoming of the 
information that can make it misleading, such as lack of currency or authority. 




The following questions helped to shape the design of this study: 



• What strategies identify effectively Web misinformation problems and devices? What 
.. clues (or “triggers”) users to the presence of misinformation? What specific triggers are 

associated with specific devices? 

• What characteristics commonly identify “reputable” or “reliable” Web documents? 

• What is the nature of Web misinformation? What devices do Web authors use to 

purposely misinform? How do these devices differ from those used in traditional 

communication, if at all? 

• How can the strategies used by sophisticated adults be adapted for younger information 

users? 

METHOD 

Data collection consisted of Web document selection and downloading, interviews, and 
observations of expert participants as they explored authentic Web misinformation documents. The project 
began with the location, downloading, and careful examination of 23 Web misinformation documents 
containing misinformation. Because it is difficult to escape the effects of personal bias in a study of this 
nature, A specific criterion was used during document selection. This selection criterion required that a 
document contain one or more of the devices of misinformation listed in previous work (Fitzgerald, 1997). 
(Still, it is necessary to admit that my personal beliefs ran strongly against the information contained in 
these documents.) Theoretically, information may contain devices of misinformation but still be valid. In 
general, however, authors of strong positions do not need to resort to such devices and avoid them because 
they lead to challenges from discerning readers. Therefore, misinformation devices are a fairly reliable 
means of identifying suspicious literature. 

From these 23 documents, two were chosen for use in this particular project. The first document, 
which claimed that the Holocaust is a hoax and, contained a rich cataloging list of misinformation devices. 
The other document, a report attempting to legitimize parapsychology to support the commercial psychic 
service operating out of the same server, contained few devices but several major problems. Neither 
document was at all believable, in the researcher’s opinion. Several other scholars and all participants in the 
study confirmed this assessment. 

PROCEDURE 

The hour-long audiotaped interviews and observations took place in a private office equipped with a 
networked Windows computer loaded with Netscape 3.0. Each session consisted of three phases: pre- 
interview, interactive evaluative task session, and post-interview. To begin, participants answered questions 
about pre-existing factors such as education, technological expertise, and bias about document topics. Next, 
they read each document in turn, followed think-aloud protocols as they read and answered assessment 
questions at completion. At the end of the session, the researcher debriefed each participant about the 
misinformation they had seen and answered any questions that occurred about the procedure. 

The interactive task portion allowed participants to freely browse the two sample Web documents 
and simultaneously describe their thoughts. When the participant fell silent, or gave responses requiring 
further explanation, the researcher interrupted with probing questions. Participants were encouraged to talk 
as much as possible in a stream-of-consciousness manner. In addition, the researcher placed no limits upon 
the amount of time spent reading each document but explicitly stated that participants could read as little or 
as much of the document as they thought necessary. At completion, summary questions sought to 
uncovered specific judgments, reasoning processes, cognitive ambiguities, reasoning, and evaluative 
criteria. 
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PARTICIPANTS 



Participants were recruited from personal colleagues at a major research university. Six graduate 
students, all with advanced Web expertise, participated in the study. Five of the students were enrolled in a 
doctoral educational technology program, while the sixth was completing a second master’s degree in 
another Field. All six had extensive experience with telecommunications and the Web. All but one were 
competent in constructing Web pages and writing HTML code. Three had participated in research projects 
directly related to Web information quality. Finally, four participants had backgrounds in the fields of 
communications, political science, advertising, and media production that gave them added expertise in 
media literacy. In sum, these participants were adequately qualified as Web information experts. Two were 
male and four were female. Ages ranged from 28 to 47. 

It is important to consider pre-existing biases in relation to the subject matter of the two documents, 
the Holocaust and parapsychology. Without exception, participants believed in the reality of the Holocaust! 
Given the sensitive nature of the material to be examined, it was determined in advance that no participants 
were Jewish. Likewise, no participant believed strongly in parapsychology, although each one 
spontaneously indicated that they were open to its possibilities. It is quite possible that the summative 
evaluations of each participant are due as much or more to these pre-existing biases than to their evaluative 
strategies (Klayman & Ha, 1987; Lord, Ross, & Lepper, 1979). In the researcher’s opinion, this limitation 
has little detrimental effect on the validity of the strategies voiced by each participant for two reasons. First, 
participants’ verbalizations clearly demonstrate active strategy use. Second, it is also clear that four 
participants experienced mild disequilibrium when presented with provocative arguments and used, in part, 
evaluation strategies to relieve this disequilibrium. 

ANALYSIS 

The Web documents were analyzed thoroughly before interviewing began. This analysis prompted 
several probing questions asked after participants made their initial judgments. After three interviews, the 
researcher transcribed them and assessed methods and results. Because the interviews seemed to yield 
useful data, three more interviews were scheduled. Interview questions underwent minor revision during 
this assessment 

After the completion and transcription of the interviews, all data, including interview transcripts, 
think-aloud transcripts, and observation notes, using open coding techniques, were analyzed. Next, the Web 
documents were reanalyzed after having reexamined the list of misinformation devices. During the second 
analysis, comparisons and parallels were drawn between devices found in the document, devices 
recognized by each participant, the triggers alerting participants to each device, and voiced evaluation 
strategies. Finally, member checks with four participants assured that strategies extracted from the 
transcripts matched participant recollections. In addition, a peer analyst, also an information expert, verified 
selected conclusions in a peer debrief. Trustworthiness of this data is further bolstered by the fact that most 
strategies appeared in the think-alouds of more than one participant. 

FINDINGS 

This study yielded three major results. First, the misinformation presented fooled no participant in 
an overall way. Second, as hoped, participants revealed valuable information evaluation strategies as they 
voiced their thoughts about the material they were reading, valuable strategies were revealed. Finally, no 
participant noticed all of the devices used in the documents. In several cases, participants were misled to a 
minor degree by a particular device. This report focuses on the second finding. The third finding requires 
deeper analysis and more data collection, and will serve as the focus of a later report. 

A summary of the misinformation devices and problems found, their definitions, associated 
triggers, and strategies used by participants to detect them is presented in Table 1. As they described their 
strategies, participants often contributed valuable insights gathered through experience. One new 
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misinformation device appeared through this sharing of collective wisdom. This device, dubbed the 
“circular reference” by one participant, Rachel, is native to the Web and allows the author to quote herself 
or himself without seeming to do so. In both misinformation documents examined in this study, authors 
included numerous links to other documents on the same server. When followed, these links usually led to 
documents by the same author. Several participants pointed out this device, and declared that they seldom 
gave much credence to references to documents residing on the same server. Another valuable tip 
contributed by Paul is to visit the front end of the server by deleting the end of the URL in Netscape's 
“open” dialog box and pressing “return.” Visiting the front end in this manner can help to establish useful 
facts about authority and purpose through identification of the sponsoring organization. 

Participants shared several unique terms for triggering mechanisms. Henry called them “red flags.” 
Rachel described the trigger process as “danger, danger, a little robin sings.” In Table 1, where available, 
interesting participant phrases describe particular triggers. It is not always possible to associate a trigger 
with a specific device through the verbalizations of the participants, a frustrating artifact of this particular 
interview protocol. Finally, Table 1 lists strategies used by participants to uncover devices successfully. 
Several of these strategies are extremely broad-spectrumed, such as “use checklist.” 

Three evaluation “styles” were evident in addition to one overall trait. Two participants, Rachel and 
Paul, showed numerous signs of using a “checklist” approach. The primary marker of this approach was 
the naming of features sought before their appearance in the text. A superficial survey of the document 
before reading began also characterized this approach. One participant, Linda, seemed to use a more 
affective approach. Affective terms characterized her reaction, and she discussed the “tone” and the 
underlying emotions of the document. Interestingly, her approach has merit because she found more, 
different devices than any of the other participants. Another approach seemed to be the “global” approach. 
Henry described his approach with this term, saying that he weighed an entire document and tried to 
balance the good against the bad to arrive at an overall evaluation. A final trend that seemed evident over all 
six participants is that the presence of a single problem or device is extremely damaging to the credibility of 
the document. 

DISCUSSION 

Limitations 

This project did not yield comprehensive results for three reasons. First, the documents chosen 
contain a limited set of devices. Thus, the list of strategies generated is not comprehensive for all devices of 
misinformation. Second, this study did not accomplished data saturation in six interviews, and thus more 
interviews should be conducted with additional participants and different documents. Finally, the 
participants in this study were aware that they would be encountering misinformation during the interactive 
Web task. The literature indicates that people significantly detect more misinformation when warned to 
search for it, but often accept it passively otherwise (Baker, 1979; Belli, 1989; Highhouse & Bottrill, 1995). 
Markman (1979) and Markman and Gorin (1981) found a similar result with children. This forewarned 
condition was designed into the study, in order to isolate evaluative strategies from the problem of whether 
or not participants would recognize misinformation in the unprimed condition. Therefore, this study does 
not address a chief concern about misinformation. 

Another obvious limitation is that findings based on data gathered from adults will be applied to 
children. However, at least one theorist asserts that the reasoning processes of children are essentially no 
different from that of adults, and children's reasoning errors are largely due to their lack of contextual 
knowledge (Carey, 1985). It is therefore reasonable to attempt a transfer of adult strategies to children. 
Evaluation of the outcomes of an instructional program based upon the results of this study may shed light 
upon Carey's theory. 

Interpretation 

Despite the limitations of this study, the results are useful. The data accomplished the primary 
purpose of strategy identification. Further, practical information contributed by the participants as well as 
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information about their evaluative styles may enhance the application of evaluative strategies in K-12 
settings. 

The checklist approach is an excellent and easily implemented starting point for information 
evaluation. Several good checklists are available (Schrock, 1996a, 1996b, 1996c; Wilkinson, Bennett, & 
Oliver, in press et al; Quinlan). However, many of the strategies used by participants in this study do not fit 
easily into the “yes, it's here/no, it's not here checklist approach. Therefore, the checklist heads a list as one 
of several strategies, all of which should be used in evaluation (Table 2). The emotional approach merits 
attention also, because sensitivity to the emotions expressed in a document may reveal devices and 
problems like stereotyping, bias, and emotional manipulation. The other evaluative style, global evaluation, 
and the tendency of the participants to discount an entire document on the basis of a single problem do not 
seem appropriate for K-12 application. 

Hemon (1995) asserts that information on the Internet is “no better or worse” than information in 
any other medium (p. 136). This notion is intuitively countered by the probability that misinformation will 
continue to flourish due to the lack of gatekeepers on the Web. In corroboration of this theory, participants 
in this study all remarked about the high percentage of low-quality information on the Web. Rachel 
expressed the gatekeeper issue in these words; The main difference is, with a book you know there had to 
be an editor at some point.. [on the Web] you have to be your own editor.” Chris expressed concerns about 
how easily electronic information can be altered, and raised the issue of political revision similar to that in 
Orwell s classic novel 1984 (1949). Linda talked at length about differences in cognitive reception of 
information between the Web and other media; 

If you've got something that's printed. .you've got to do something. You have to throw it in 
the trash, you have to put it back, it's in your hand. So it's a little bit more tangible.. .When 
you re on the Web, you just [click] and it's gone. So much easier, so much faster, so much 
more immediate... In a way maybe that's a good thing because you're out of there really fast; 
but maybe that's a bad thing because as you're carrying this back to throw it in the trash or 
put it away you’re reaffirming the fact that this is not the right information. 



This former primary school teachers observation resonates with the concrete operations theories of Piaget 
(1948). - 6 

The participants in this study constructed their strategies over years of practice and study. These 
strategies, carefully selected and adapted according to pedagogical principles, can be taught to children. 
Drawing upon the researcher’s own K-12 teaching experience and the writings of educational theorists, 
suggested teaching strategies are listed below. 

APPLICATION 

Teaching evaluative strategies to children is challenging. Children cannot be instructed to search for 
abstract constructions like bias and logical fallacies without extensive preparation. Children of elementary 
school age are also uncomfortable with the disequilibrium caused by ambiguity, a necessary 
accompaniment to evaluative thinking (Piaget, 1948). 

Another issue to be addressed is a philosophical one. At what age is it appropriate to teach children 
that people sometimes lie? Many parents and teachers believe that children should learn to trust adults and 
obey authority, and that discussions about lying will undermine this teaching. On the other hand, children 
witness untruths, fiction, and fantasy every day in many different media. More ominously, people eager to 
exploit the naive and the young through commercial and criminal means stalk the Internet. This troubling 
issue is far beyond the scope of this report, but teachers and parents must consider it before applying 
interventions for misinformation detection. 

Specific strategies suitable for K-12 students appear in Table 2. Because of the complexity of 
several of these strategies, recommended grade levels for introduction are included. In addition, many of 
these strategies occur in varying levels according to the demands of the text. The teacher is the best judge of ’ 
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when and how to apply differing levels of difficulty. Most of the strategies, especially argumentation, 
should be taught over a span of years. 

It is vitally important that information literacy skills in general and these strategies in particular be 
taught in the context of subject matter material (Callison, 1993). Luckily, critical thinking strategies fit well 
with most content areas. Few topics could be more boring or incomprehensible to children than critical 
thinking or argumentation taught out of context. The best approach is to choose a subject area of current, 
controversial interest to the students in a given class, and integrate the suggested strategies into a unit about 
that topic. For example, a current educational controversy is whether or not school children should wear 
uniforms. People with fervent opinions speak hotly on both sides of this issue, and students feel 
powerlessly caught between. Opinions on this topic appear in newspapers, broadcast media, the Web, and 
Internet discussion forums. Students could explore this issue through all of these media. In the course of 
doing so, misinformed opinions and false information are bound to surface, and the strategies listed in 
Table 2 will necessarily come into play. This research should culminate in some public forum such as a 
debate or a special issue of the school newspaper. Projects such as these require a great deal of planning and 
time on the part of teachers and media professionals, but Dewey (1915) recommends them as an extremely 
most effective type of learning. 

CONCLUSION 

The series of qualitative interviews reported in this paper explored questions relating to Web 
misinformation and strategies for detecting such misinformation. Expert Web users served as a source of 
strategies for others who can benefit from their experience. Suggestions for adapting, applying, and 
teaching these strategies in K-12 contexts were made. Although these strategies may seem straightforward 
at first glance, the pedagogical and philosophical issues involved in teaching them to children include 
complicated pedagogical and philosophical issues. Nevertheless, educators must address these issues to 
prepare the students of today for their futures as information consumers. 

This area of inquiry provides many opportunities for future research. Interviews of more experts 
with different documents containing different devices should reveal new strategies keyed toward specific 
devices. In addition, a repertoire of several strategies may better serve the different learning styles 
represented among different individuals. Two other vital but inadequately explored issues involve the 
spontaneous application of critical thinking and the long-term effects of misinformation upon memory. 
Presently, we know very little about why people decide to use evaluative skills in a given situation, provided 
that they have such skills. While psychologists have studied misinformation effects in respect to memory, 
few studies have addressed misinformation effects in an educational sense. The World Wide Web presents 
an excellent medium through which to study all of these issues. 

While we know few facts about the effects of Web misinformation upon people, it seems likely 
that more misinformation will be published on the Web than in any other medium except for spoken 
communication unless some agency intervenes and begins to “police” the Web. As the only truly 
democratic medium except for spoken communication, such policing would spoil a valuable public 
resource. As it stands, educators should view the Web as an opportunity to further the vital critical thinking 
agenda. In the meantime, it is imperative that scholars and educators learn more about the effects of 
misinformation. 

Debate continues over how the educational establishment can best foster the intertwined skills of 
critical thinking and information literacy. This research should contribute to educational efforts by 
describing strategies that successfully detect misinformation. Further, this exposure and analysis of Web 
misinformation samples may awaken users to the importance of critical evaluation and help to encourage 
the spontaneous application of critical reasoning. 
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